Papers with Softmax mechanism
SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMs (2026.findings-acl)
Copied to clipboard
Jie Sun, Yu Liu, Lu Han, Qiwen Deng, Xiang Shu, Yang Xiao, Lintao Ma, Xingyu Lu, Jun Zhou, Pengfei Liu, Jiancan Wu, Xiang Wang
| Challenge: | Existing large-scale large-context models suffer from performance degradation when processing long numerical sequences. |
| Approach: | They propose a framework to mitigate attention dispersion by strategically inserting separator tokens into the model to recalibrat attention to local segments while preserving global context. |
| Outcome: | The proposed framework improves accuracy and reduces inference token consumption by 16.4% on 9 widely-adopted LLMs. |